Autoprocess


An in-house script called Autoprocess has been develloped at the CMCF. It generates appropriate input files for, and runs XDS. It uses POINTLESS to identify the space group from systematic absences and BEST to create a data collection strategy. A new directory is created that contains the XDS input and output files as well as .mtz, .shelx and .cns reflection files which can be used directly in your favourite structure solution/refinement programs.

Running Autoprocess

Run Autoprocess either from the MxDC data collection software or from a Data Processing terminal command line.

In MxDC running Autoprocess is a very easy and seamless process. Simply go to the Processing Results tab, select the dataset and click either Process or Screen, the anomalous data or MAD data checkboxes being optional. Processing in MxDC will automatically upload the results to MxLIVE, the results being associated with the sample that has been mounted. Additional information is available in the MxDC Processing Results tab section. 

To run Autoprocess from the command line, double-click the Data Processing icon.

Input

Start Autoprocess by typing "autoprocess" followed by any options and representative image file(s). Some options include -s (screen/characterize crystal and obtain a collection strategy), -a (to indicate anomalous data) and -m (MAD data). Images do not have to be sequential.

    example: process a native dataset

    autoprocess any_image_file.img

    example: process an anomalous dataset

    autoprocess -a any_image_file.img

    example: obtain strategy for a native data collection from 5 or more images

    autoprocess -s any_image_file.img

    example: obtain strategy for an anomalous data collection from 5 or more images

    autoprocess -as any_image_file.img

    example: processing a MAD dataset  

    autoprocess -m any_peak_file.img any_infl_file.img any_remo_file.img

    example: merging two or more datasets  

    autoprocess set1_image_file.img set2_image_file.img ... 

    A full list of options is displayed by typing "autoprocess" with no options:

    autoprocess [options] /path/to/set1.img /path/to/set2.img ... /path/to/setn.img

    options:    
         --mad, -m : Process each set, scale together and generate separate reflection files.
         --screen, -s : Process a few frames and characterize crystal from each set.
         --anom, -a : Process with Friedel's law False.
         --backup, -b : Backup previous output directory if it exists.
         --prefix=p1,p2,p3 : Comma separated list of prefixes to use for output files.
                   Default is first part of image name.
                   Prefix order should correspond to the order of the data sets.
                        For example for MAD data, use --prefix=peak,infl,remo.
         --dir=/path : Directory to store processed results.
                   Default is to create a new one in the current directory.
         --inputs, -i : Generate XDS.INP only and quit.
         --help, -h : Display this message.
         Default (no option) : Process each set, scale together and merge into one reflection file.

    data sets:
         Each data set can be represented by any frame from that set. 

Output

Autoprocess by default creates a new directory that contains the XDS input and output files. The new directory name ends in either -proc (data processing output) or -scrn (crystal screening/strategy). Data processing directories will also contain .mtz, .shelx and .cns reflection files. Processing summaries are output as a text file named 'process.log' and a 'report' folder which contains a web browser-compatible 'index.html' file as well as several .png image files.

    To view the process.log file type

    more process.log     OR     gedit process.log 

    To view the web browser compatible report type  

    firefox report/index.html  

Modifying the Processing

There may be times when you want to specify certain parameters for Autoprocess, such as space group, resolution limit or image range. This is done by using one of the auto.* options which repeats a certain part of the Autoprocess procedure from that step onward, using the modified parameter(s). Follow the steps bellow:

  1. Run Autoprocess as usual, either via MxDC or manually in a Data Processing terminal.
  2. Go to the output directory created by Autoprocess (-proc, or -scrn).
  3. Type one of the following auto.* commands, followed by -h to view instructions and options for modifying the processing from that point onward:
    • auto.analyse
    • auto.inputs
      • generate XDS.INP file for running XDS manually
    • auto.integrate
      • resume, repeat or optimize a previous processing from the integration step onward, with the same or different settings for resolution, data range or Friedel's law
    • auto.report
      • generate HTML reports from a "process.json" harvest file
    • auto.scale
      • resume or repeat a previous processing from the scaling step onward, with the same or different settings for resolution range or Friedel's law
    • auto.strategy
      • resume or repeat previous processing from the strategy determination step onward with the same or different settings for resolution and Friedel's law
    • auto.symmetry
      • resume or repeat a previous processing from the spacegroup determination step onward, with the same or different settings for resolution range or spacegroup

example: to modify the processing to use spacegroup P2(1)2(1)2(1), run Autoprocess then within the -proc directory, type:

auto.symmetry --spacegroup=19 OR auto.symmetry -g 19

example: use a different data range for integration

auto.integrate --frames=20-120

example: set high resolution limit for scaling to 2.2

auto.scale --res=2.2

Now that you have processed your data in a new spacegroup or with new resolution cutoffs, you may wish to upload the new result to MxLIVE so that your colleagues back home can view your work. This can easily be accomplished by typing:

upload-report process.json

Advanced Usage

Besides the reflection files and reports, Autoprocess creates numerous other files, including

  • image files (.cbf)
  • XDS input files (.INP)
  • XDS output files (.LP)

Viewing .cbf Files

You may view the .cbf files with imgview or xds-viewer. For example, to view spots on the last frame processed, type

imgview FRAME.cbf     OR     xds-viewer FRAME.cbf

Using XDS Input Files

There may be instances when you want to run XDS manually. You can view a full list of XDS input parameters on the XDS homepage.

  • In these cases, XDS may be run directly by typing xds_par, but suitable input files must be prepared before it can be used.
  • It is recommended to first run Autoprocess on your image files to obtain appropriate input files, then modify them accordingly. You can also create XDS.INP easily by typing
  • auto.inputs  any_image_file.img
  • An example of an input file appears below:
    !-XDS.INP-----------File generated by autoprocess
    JOB=ALL
    MAXIMUM_NUMBER_OF_PROCESSORS=16
    MAXIMUM_NUMBER_OF_JOBS=2
    MINUTE=0
    !-------------------Dataset parameters-------
    X-RAY_WAVELENGTH=0.97934
    DETECTOR_DISTANCE=150.0
    OSCILLATION_RANGE=1.00
    SPACE_GROUP_NUMBER=75
    UNIT_CELL_CONSTANTS= 77 77 38 90 90 90
    !REIDX= 0 0 -1 0 0 -1 0 0 -1 0 0 0
    NAME_TEMPLATE_OF_DATA_FRAMES=images/Gd_Lysozyme_1_E0_????.img TIFF
    DATA_RANGE=1 190
    SPOT_RANGE=1 5
    !-------------------Beamline parameters-----
    NX=3072 NY=3072
    QX=0.07324 QY=0.07324
    ORGX=1512 ORGY=1537

    DETECTOR=CCDCHESS
    MINIMUM_VALID_PIXEL_VALUE=0
    OVERLOAD=65000
    TRUSTED_REGION=0.02 1.35
    VALUE_RANGE_FOR_TRUSTED_DETECTOR_PIXELS= 6000 30000
    ROTATION_AXIS= 1.0 0.0 0.0
    INCIDENT_BEAM_DIRECTION=0.0 0.0 1.0
    FRACTION_OF_POLARIZATION=0.95
    POLARIZATION_PLANE_NORMAL= 0.0 1.0 0.0
    DIRECTION_OF_DETECTOR_X-AXIS= 1.0 0.0 0.0
    DIRECTION_OF_DETECTOR_Y-AXIS= 0.0 1.0 0.0
    AIR=0.00033

    !-------------------File generated by autoprocess

Notes

  • Autoprocess is not available off-site at this time.
  • Autoprocess was formerly also known as autoxds, a name that has since been adpoted at SSRL.
  • Although the old name may still appear in some documentation, it is not to be confused with anything in use at SSRL.