Determining matrix from linear transformation

In the week 3 video linear transformation as matrices, after the 1:12 mark, the instructor explains that we only need (1,0) and (0,1) vector to find the matrix corresponding to the transformation. We know that (1,0) → (3.-1) and (0,1) → (2,3). And so our matrix will have the columns:

[3,2]
[-1,3]

Obviously this works but my question is that how did we know to place 3,-1 in the first column and 2,3 in the second?

(3,-1) corresponds to the transformation of (1, 0) which is corresponds to x-axis and the other one with the y axis. So you start the transformation matrix with x then y then z in the same way coordinates are mapped!