Every DDS system requires necessarily an analogue band pass filter behind this DAC in order to select the frequency band of interest out of the generated spectrum. Usually the lowest image of the spectrum is filtered out and so the band pass filter is usually just a low-pass filter. Its filter parameters determine the maximum usable output frequency over the master clock frequency. Practically achievable output frequency bandwidths go up to about 40% of the master clock frequency.
The task of the anti aliasing filter is to pass all frequencies of the desired frequency band and to stop all others. A typical DDS generator would select the lowest image of the output spectrum and employ a low pass filter for this task. The desired performance of the low pass filter would therefore be 0 dB attenuation below the Nyquist frequency and maximum attenuation above. Such a behaviour is impractical and so one has to make compromises. For radio applications, one could probably accept a passband ripple of 3 dB and a stopband attenuation in the order of the DAC's total harmonic distortion. For instruments like a sweep generator, the requirements at least for passband ripple and phase are certainly a little bit higher.
The NCO must avoid frequencies that fall into the transition band, because these frequencies and their aliases will either be too much attenuated to be useful or not sufficiently attenuated to be harmless.
Useful anti-aliasing filter types include the Butterworth and Chebychev filters. Both have a rather steep cutoff that results in fairly low filter orders. The Butterworth filter is flat in the passband but not as steep in the transition region as the Chebychev filter. The downside of the Chebychev filter is its passband ripple, which can be made arbitrarily small, however, at the expense of its cutoff. It seems that a Chebychev filter is sufficient for radio applications, especially when operating on rather small frequency bands and when an analogue gain control stage follows. For instruments like a sweep generator, the Butterworth filter with higher filter order is probably better suited due to its flat passband.
For achieving the desired bandwidth of 40% of the master clock frequency, one can employ a 7th order Chebychev filter or a 9th or 11th oder Butterworth filter as seen in the pictures below. The filters have been syntesized with the Filtry and simulated with APLAC.