Canadian Macromolecular Crystallography Facility

AutoProcess


An in-house software program called AutoProcess has been developed at the CMCF. It automatically generates appropriate input files for, and runs XDS. It uses POINTLESS to identify the space group from systematic absences and BEST to determine a data collection strategy. All generated input and output files are saved in a new directory, ncluding .mtz, .shelx and .cns reflection output files which can be used directly in your favourite structure solution/refinement programs.

Running Autoprocess

Run Autoprocess either from the MxDC data collection software or from a Data Processing terminal command line.

In MxDC running AutoProcess is a very easy and seamless process. Simply go to the Processing Results tab, select the dataset(s), select the processing options and click Run. Processing in MxDC will automatically upload the results to MxLIVE, associated with the sample on which the data were collected. Additional information is available in the MxDC Processing Results tab section. 

To run AutoProcess from the command line, it is required to first log-in to one of our high-performance computing nodes of our Data Processing cluster.  You can do that by double-clicking on the Data Processing icon on the Desktop.

Input

Note that AutoProcess will create a directory for output files within the current working directory. Therefore, change the directory appropriately before issuing the commands.

Start AutoProcess by typing "auto.process" followed by any options and representative image file(s). Some options include -s (screen/characterize crystal and obtain a collection strategy), -a (to indicate anomalous data) and -m (MAD data). Images do not have to be sequential.

example: process a native dataset

auto.process any_image_file.img

example: process an anomalous dataset

auto.process -a any_image_file.img

example: obtain strategy for a native data collection, requires at least 4 degrees of data to produce reliable results

auto.process -s any_image_file.img

example: obtain strategy for an anomalous data collection

auto.process -as any_image_file.img

example: processing a MAD dataset  

auto.process -m any_peak_file.img any_infl_file.img any_remo_file.img

example: merging two or more datasets  

auto.process set1_image_file.img set2_image_file.img ... 

A full list of options is displayed by typing "autoprocess" with no options:

auto.process [options] /path/to/set1.img /path/to/set2.img ... /path/to/setn.img

options:    
     --mad, -m : Process each set, scale together and generate separate reflection files.
     --screen, -s : Process a few frames and characterize crystal from each set.
     --anom, -a : Process with Friedel's law False.
     --backup, -b : Backup previous output directory if it exists.
     --prefix=p1,p2,p3 : Comma separated list of prefixes to use for output files.
               Default is first part of image name.
               Prefix order should correspond to the order of the data sets.
                    For example for MAD data, use --prefix=peak,infl,remo.
     --dir=/path : Directory to store processed results.
               Default is to create a new one in the current directory.
     --inputs, -i : Generate XDS.INP only and quit.
     --help, -h : Display this message.
     Default (no option) : Process each set, scale together and merge into one reflection file.

data sets:
     Each data set can be represented by any frame from that set. 

Output

AutoProcess by default creates a new working directory that contains the XDS input and output files. The new directory name depends on the requested mode of operation and will be named either "proc-native" (native data processing output) or "proc-screen"(crystal screening/strategy), or "proc-mad" (MAD data processing), or "proc-merge" (merging multiple datasets), or "proc-anom" (anomalous data processing). Alternatively, a specific directory name can be forced by adding the "--dir=/required/output/directory" option to the command. For multi-dataset processing such as MAD or Merging, sub-directories will be created within the working directory described above, for individual datasets. The output .mtz, .shelx and .cns reflection files can be found in the top-level working directory. This directory will also contain output as report files in HTML (report.html) and TEXT (report.txt) formats, summarizing the results. The HTML file can be viewed using a browser. TEXT files, and various other .log files can also be viewed using your favorite text editor/viewer such as "gedit" or "more".

To view the web browser compatible report type  

firefox report.html  

Modifying the Processing

AutoProcess in addition to providing the "auto.process" command, also provides additional commands to be used at times when you want to specify certain parameters, such as space group, resolution limit or image range. The extra commands work by starting AutoProcess from a given step within the processing chain and can be used to repeat a given step. Each of the commands, while starting from a different step,  will run all subsequent steps to the end, using the modified parameter(s).

All the commands provided by AutoProcess, including the main "auto.process" command, accept a "-h" or "--help" option which prints out detailed help about available options and examples.

Since the extra commands can only be used to modify a previously run AutoProcess session, it is required to change into the top-level AutoProcess working directory before running any of these commands.

  • auto.process -- When run from within a working directory without any parameters, it means try to continue an interrupted session from where it was interrupted.
  • auto.inputs -- Generate XDS.INP file for running XDS manually
  • auto.integrate -- Repeat or optimize a previous processing from the integration step onward, with the same or different settings for resolution, data range or Friedel's law
  • auto.report -- Generate HTML and TEXT report files report.html, report.txt
  • auto.scale -- Repeat a previous processing from the scaling step onward, with the same or different settings for resolution range or Friedel's law
  • auto.strategy -- Repeat previous processing from the strategy determination step onward with the same or different settings for resolution and Friedel's law
  • auto.symmetry -- Repeat a previous processing from the spacegroup determination step onward, with the same or different settings for resolution range or spacegroup

example: to modify the processing to use spacegroup P2(1)2(1)2(1), change into the working directory and type:

auto.symmetry --spacegroup=19 OR auto.symmetry -g 19

example: use a different data range for integration

auto.integrate --frames=20-120

example: set high resolution limit for scaling to 2.2

auto.scale --res=2.2

Advanced Usage

Besides the reflection files and reports, XDS creates numerous other files, including

  • image files (.cbf)
  • XDS input files (.INP)
  • XDS output files (.LP)

Viewing .cbf Files

You may view the .cbf files with imgview or xds-viewer. For example, to view spots on the last frame processed, type

imgview FRAME.cbf     OR     xds-viewer FRAME.cbf

Using XDS Input Files

There may be instances when you want to run XDS manually. You can view a full list of XDS input parameters on the XDS homepage.

    • In these cases, XDS may be run directly by typing xds_par, but suitable input files must be prepared before it can be used.
    • It is recommended to first run AutoProcess on your image files to obtain appropriate input files, then modify them accordingly. You can also create XDS.INP easily by typing
auto.inputs  any_image_file.img
  • An example of an input file appears below:
    !- XDS.INP ----------- Generated by AutoProcess
    JOB=   ALL ! CORRECT
    !------------------- Dataset parameters
    X-RAY_WAVELENGTH=  0.97934
    DETECTOR_DISTANCE= 200.0
    STARTING_ANGLE=    100.0
    STARTING_FRAME=    100.0
    OSCILLATION_RANGE= 0.50
    NAME_TEMPLATE_OF_DATA_FRAMES=/users/school/insulin/insulin_???.img
    FRIEDEL'S_LAW= TRUE
    DATA_RANGE=    1 120
    DELPHI=    8.00
    SPOT_RANGE=    1 8
    SPOT_RANGE=    91 98
    BACKGROUND_RANGE=  1 10
    SPACE_GROUP_NUMBER=    197
    UNIT_CELL_CONSTANTS=   78.000 78.000 78.000 90.000 90.000 90.000
    !----------------- Beamline parameters
    DETECTOR= CCDCHESS
    NX=3072    NY= 3072
    QX=0.07324 QY=0.07324
    ORGX= 1516  ORGY= 1536
    SENSOR_THICKNESS= 0.000
    MINIMUM_VALID_PIXEL_VALUE= 1
    OVERLOAD= 65535
    STRONG_PIXEL=     6
    MINIMUM_ZETA= 0.05
    TRUSTED_REGION=0.00 1.414
    TEST_RESOLUTION_RANGE= 50.0 1.0
    TOTAL_SPINDLE_ROTATION_RANGES= 90 360 30
    STARTING_ANGLES_OF_SPINDLE_ROTATION= 0 180 15
    VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS= 6000 30000
    INCLUDE_RESOLUTION_RANGE=50.0 0.0
    ROTATION_AXIS= 1.0 0.0 0.0
    INCIDENT_BEAM_DIRECTION=0.0 0.0 1.0
    FRACTION_OF_POLARIZATION=0.95
    POLARIZATION_PLANE_NORMAL= 0.0 1.0 0.0
    DIRECTION_OF_DETECTOR_X-AXIS= 1.000 0.000 0.000
    DIRECTION_OF_DETECTOR_Y-AXIS= 0.000 1.000 0.000
    !----------------- Extra parameters