First, it should be noted that a mecanum drive system won't behave like a tank drive exactly. The behavior of mecanum wheels will prevent this; driving one side forward and the other forward with less power won't cause tank-like movement. However, movement in this
style is still possible.
To properly use a mecanum drive system, you need to express the heading as an angle and magnitude (in polar coordinates) as well as a rotational scalar. The first thing to do is to convert both of your Y-axes to a set of polar coordinates; the difference between the two should become the rotational scalar. Then, you factor the triggers into the rotational scalar.
Here's what I'd try:
Code:
y_max=max(y1,y2)
y_min=min(y1,y2)
mag = S1*(y1+y2)/2
theta = arctan((y_max-y_min)/y_min)
rot = S2*(((y_max-y_min)/y_max)+S3*triggers)
Where y_max is the max of the two ys, y_min is the min, mag is the magnitude in polar coordinates, theta is the direction of movement, and rot is the rotational scalar for the robot. S1, S2, and S3 are constants. Using these values as well as a proper mix of sines and cosines, you can run a mecanum drive. Hopefully this should do what you'd like; conceptually it works in my mind, at least.
Hope it helps!