Get 3d Point In Space Using 2d Point In Image In Python Opengl

Im trying to simulate a depth camera in a room, my camera is able to move and rotate in the world and the room is simulated as a 3d cube around the (0,0,0) At the click of a button

Solution 1:

To find the world position of a point on the viewport, you have to know the depth value of the point.

The x and y screen position and the depth have be transformed to normalized device coordinates in the range [-1, 1]. For this the viewport rectangle has to be known:

ndc = [2.0* x/vp_width - 1.0, 1.0 - 2.0*y/vp_height, depth*2.0 - 1.0]; 

The normalized device space coordinate has to be transformed by the inverse projection matrix to the view space (Finally a perspective divide has to be performed).

With the inverse view matrix, the view space coordinate can be transformed to world space.

gluUnProject does all this for you, but you have to know the depth of the fragment. The depth of a fragment can be read by glReadPixels:

# get mouse position
x, y = pygame.mouse.get_pos()

# get the fragment depth
depth = glReadPixels(x, y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT)

# get projection matrix, view matrix and the viewport rectangle
model_view = np.array(glGetDoublev(GL_MODELVIEW_MATRIX))
proj = np.array(glGetDoublev(GL_PROJECTION_MATRIX))
view = np.array(glGetIntegerv(GL_VIEWPORT))

# unproject the point
point = gluUnProject(x, y, depth, model_view, proj, view)
print( point )

---

Note, you have to enable the Depth Test otherwise the depth buffer will not be set. This also gives the benefit, that polygons at the front cover the polygons "behind" them:

glEnable(GL_DEPTH_TEST)
Cube()

---

Of course the projection matrix and the model view matrix have to be proper set, when the vlues are read by glGetDoublev(GL_PROJECTION_MATRIX) respectively glGetDoublev(GL_MODELVIEW_MATRIX).

This means that the reading of the view matrix should be done after setting it:

glPushMatrix()
glLoadIdentity()
glTranslatef(tx, ty, tz)
glRotatef(ry, 0, 1, 0)
glRotatef(rx, 1, 0, 0)

glMultMatrixf(view_mat)
glGetFloatv(GL_MODELVIEW_MATRIX, view_mat)

model_view = np.array(glGetDoublev(GL_MODELVIEW_MATRIX))

Note, if for the 4th parameter (model) of gluUnProject is used the identity matrix, then gluUnProject does not calculate world coordinates, but it calculates view coordinates.