Interesting, I can reproduce the problem with both vptestmb and vptestmw.
Adding option evex:1 and arch:avx to recognize avx512 opcodes.
.xmm
option arch:avx
option evex:1
kmovd ecx,k0 ;ok
vptestmd k0, ymm15, xmmword ptr [rbx] ;ok
vptestmq k0, ymm15, xmmword ptr [rbx] ;ok
vptestnmd k0, ymm15, xmmword ptr [rbx] ;ok
vptestnmq k0, ymm15, xmmword ptr [rbx] ;ok
In reswords.c line 677 changing from VX_NND to VX_HALF, the opcode is recognized and successfully assembles.
VX_LL|VX_L|VX_HALF, /* VPTESTMB */
VX_LL|VX_L|VX_HALF, /* VPTESTMW */
VX_LL|VX_L|VX_HALF, /* VPTESTMD */
VX_LL|VX_L|VX_HALF, /* VPTESTMQ */
VX_LL|VX_L|VX_HALF, /* VPTESTNMB */
VX_LL|VX_L|VX_HALF, /* VPTESTNMW */
VX_LL|VX_L|VX_HALF, /* VPTESTNMD */
VX_LL|VX_L|VX_HALF, /* VPTESTNMQ */
vptestmw k0, ymm15, ymmword ptr [rbx] ; 62 F2 85 28 26 03