Single-Modality Techniques

CT

Figure 1: Raycasted bonestructure of the CT volume.
Computed tomography (CT) is a medical imaging method employing tomography created by computer processing.Digital geometry processing is used to generate a three-dimensional image of the inside of an object from a large series of two-dimensional X-ray images taken around a single axis of rotation.[1]

PET

Figure 2: The maximum intensity projection of the PET volume.

Positron emission tomography (PET) is a nuclear medicine imaging technique which produces a three-dimensional image or picture of functional processes in the body. The system detects pairs of gamma rays emitted indirectly by a positron-emitting radionuclide (tracer), which is introduced into the body on a biologically active molecule. Images of tracer concentration in 3-dimensional or 4-dimensional space (the 4th dimension being time) within the body are then reconstructed by computer analysis. In modern scanners, this reconstruction is often accomplished with the aid of a CT X-ray scan performed on the patient during the same session, in the same machine. [2]

MRI

Figure 3: MRI T1 raycasted.
Figure 4: MRI T2 raycasted.
Magnetic resonance imaging (MRI), or nuclear magnetic resonance imaging (NMRI), is primarily a medical imaging technique most commonly used in radiology to visualize detailed internal structure and limited function of the body. MRI provides much greater contrast between the different soft tissues of the body than computed tomography (CT) does, making it especially useful in neurological (brain), musculoskeletal, cardiovascular, and oncological (cancer) imaging. Unlike CT, it uses no ionizing radiation, but uses a powerful magnetic field to align the nuclear magnetization of (usually) hydrogen atoms in water in the body. Radio frequency (RF) fields are used to systematically alter the alignment of this magnetization, causing the hydrogen nuclei to produce a rotating magnetic field detectable by the scanner. This signal can be manipulated by additional magnetic fields to build up enough information to construct an image of the body.[3]

Multi-Modal Techniques

Technique 1: T1 or CT based on PET Intensity

Figure 5: Technique 1: T1 or CT based on PET Intensity.
With the aim of displaying the high intensity PET areas, with as much context as possible we utilize CT, MRI T1 and PET in a multimodal visualization. Areas with a high intensity PET, display the MRI T1 blended by the color of the PET. Areas with a low PET intensity use the color from the CT volume (through a transfer function displaying bone). 
vec4 T1orCTbasedonPET( vec3 pos ){
  vec4 texel = texture3D(PET, pos);
  //If the PET high enough?
  if ( texel.a > 0.6 ){
    float mriv = texture3D( MRIT1, pos ).a;
    //Make the PET color darker
    //and more transparent
    //in areas with low T1
    texel.rgb *= mriv;
    texel.a = mriv;
  }
  // if not, then use CT
  else{
    texel = texture1D(cmCT, texture3D(CT,pos).a );
  }	
  return texel;
}

Technique 2: PET MIP onto CT

Figure 6: Technique 2: PET MIP onto CT.
This technique utilizes the CT and the PET dataset. When the raycaster hits bone in the CT dataset, the raycaster starts a MIP from there on, and utilizes this as the color on that hit position.

Technique 3: T1 behind Bone colored by PET

Figure 7: Technique 3: T1 behind Bone colored by PET.
This technique renders the parts of T1 which is hidden behind bone, i.e., inside the skull. The shown T1 parts are furthermore shown using the color of PET. The following pseudocode is used to detect if we have passed through one "level" of bone. 
bool through_bone = false;
bool inside_bone = false;
// Resulting texel color
vec4 texel = vec4( 0,0,0,0 );

Raycasting Loop{
    vec4 voxel = vec4( 0,0,0,0 );
    float cta = texture1D(colormap,texture3D(CT,pos).a).a;
    if ( ! through_bone ){
        if (inside_bone){
            if (cta < 0.1 )
                through_bone = true;
        }
        else{
            if (cta > 0.1 )
                inside_bone = true;
        }
    }
    else{
        voxel = texture3D( PET, pos );
        voxel.a = texture3D( MRIT1, pos ).a;
    }    
    texel += blend( texel, voxel );
}

Technique 4: T1 and T2 Differences

Figure 8: Technique 4: T1 and T2 Differences.
Figure 9: Colormap used for difference view.
A simple raycaster that utilizes MRI T1 and T2, to create a difference view showing T1 minus T2. The colormap used in this visualization is shown in Fig. 9. 
vec4 mri_difference( vec3 pos ){
  float value = texture3D( MRIT1, pos ).a 
              - texture3D( MRIT2, pos ).a;
  //Enlarge the difference by a factor (e.g., 20)
  //to highlight the differences
  value = 20*value;
  vec4 texel = texture1D(diverging_cmap,(value+1.0)*0.5 );
  return texel;
}