> the documentation says "not allowed".
The documentation is wrong. It can be used if the model type is DNLP.
Both the GAMS user guide and the McCarl guide have exactly identical errors in their descriptions of GAMS functions. The functions trunc(x), sign(x), ceil(x), mod(x,y), floor(x) and round(x) are all allowed in DNLP models opposed to what is stated.
Note that DNLP models are treacherous. In many case you may want to consider alternative formulations such as a smooth approximation or a formulation using binary variables. Specifically the floor(x) function can be modeled as:
integer variable ifloor;
ifloor ≤ x
ifloor ≥ x - 0.9999999