基于词编码模型进行词类推和去歧视

发表于 | 分类于

由于训练词编码模型十分耗时，大多数机器学习实用主义者都是加载已经训练好的模型。本文同样是使用训练好的词向量，利用余弦相似度计算相似度，解决词类比问题（比如男人和女人类似于国王和王后）。另外，本文也修改词编码方式解决性别歧视问题。 

1
2
import numpy as np
from w2v_utils import *
Using TensorFlow backend.

本文使用50维的GolVe向量作为词向量。下面加载词编码模型。

1
words, word_to_vec_map = read_glove_vecs('model_data/glove.6B.50d.txt')

加载后得到：
- words: 单词表
- word_to_vec_map: 将单词映射到GloVe向量的字典

你已经看到了，one-hot向量无法获取两个单词是否意思相近。而GloVe向量可以提供更多有意义的信息。

余弦相似度

给定两个编码后的词向量\(u\)和\(v\), 余弦相似度定义为：

\[\text{CosineSimilarity(u, v)} = \frac {u . v} {||u||_2 ||v||_2} = cos(\theta) \tag{1}\]

其中 \(u.v\) 为两个向量之间的点积 (或内积), \(||u||_2\) 是向量\(u\)的模 (或长度) , \(\theta\) 是两个向量 \(u\) 和 \(v\)的夹角. 相似度依赖于 \(u\) 和 \(v\)的夹角. 如果 \(u\) 和 \(v\) 相似, 那么余弦相似度接近1; 如果它们不相似, 余弦相似度很小.

图 1: 两个向量的夹角度量它们的相似度 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# GRADED FUNCTION: cosine_similarity

def cosine_similarity(u, v):
    """
    Cosine similarity reflects the degree of similariy between u and v
        
    Arguments:
        u -- a word vector of shape (n,)          
        v -- a word vector of shape (n,)

    Returns:
        cosine_similarity -- the cosine similarity between u and v defined by the formula above.
    """
    
    distance = 0.0
    
    ### START CODE HERE ###
    # Compute the dot product between u and v (≈1 line)
    dot = np.dot(u,v)
    # Compute the L2 norm of u (≈1 line)
    norm_u = np.sqrt(np.sum(u*u))
    
    # Compute the L2 norm of v (≈1 line)
    norm_v = np.sqrt(np.sum(v*v))
    # Compute the cosine similarity defined by formula (1) (≈1 line)
    cosine_similarity = dot/norm_u/norm_v
    ### END CODE HERE ###
    
    return cosine_similarity
1
2
3
4
5
6
7
8
9
10
11
12
father = word_to_vec_map["father"]
mother = word_to_vec_map["mother"]
ball = word_to_vec_map["ball"]
crocodile = word_to_vec_map["crocodile"]
france = word_to_vec_map["france"]
italy = word_to_vec_map["italy"]
paris = word_to_vec_map["paris"]
rome = word_to_vec_map["rome"]

print("cosine_similarity(father, mother) = ", cosine_similarity(father, mother))
print("cosine_similarity(ball, crocodile) = ",cosine_similarity(ball, crocodile))
print("cosine_similarity(france - paris, rome - italy) = ",cosine_similarity(france - paris, rome - italy))
cosine_similarity(father, mother) =  0.890903844289
cosine_similarity(ball, crocodile) =  0.274392462614
cosine_similarity(france - paris, rome - italy) =  -0.675147930817

词类推

在词类推任务中，需要完成"a is to b as c is to ____"。一个例子是 'man is to woman as king is to queen' 。具体的说，该任务是寻找d, 使得相关的词向量 \(e_a, e_b, e_c, e_d\) 满足: \(e_b - e_a \approx e_d - e_c\). 这里使用余弦相似度计算 \(e_b - e_a\) 和 \(e_d - e_c\) 之间的相似度.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# GRADED FUNCTION: complete_analogy

def complete_analogy(word_a, word_b, word_c, word_to_vec_map):
    """
    Performs the word analogy task as explained above: a is to b as c is to ____. 
    
    Arguments:
    word_a -- a word, string
    word_b -- a word, string
    word_c -- a word, string
    word_to_vec_map -- dictionary that maps words to their corresponding vectors. 
    
    Returns:
    best_word --  the word such that v_b - v_a is close to v_best_word - v_c, as measured by cosine similarity
    """
    
    # convert words to lower case
    word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()
    
    ### START CODE HERE ###
    # Get the word embeddings v_a, v_b and v_c (≈1-3 lines)
    e_a, e_b, e_c = word_to_vec_map[word_a], word_to_vec_map[word_b], word_to_vec_map[word_c]
    ### END CODE HERE ###
    
    words = word_to_vec_map.keys()
    max_cosine_sim = -100              # Initialize max_cosine_sim to a large negative number
    best_word = None                   # Initialize best_word with None, it will help keep track of the word to output

    # loop over the whole word vector set
    for w in words:        
        # to avoid best_word being one of the input words, pass on them.
        if w in [word_a, word_b, word_c] :
            continue
        
        ### START CODE HERE ###
        # Compute cosine similarity between the vector (e_b - e_a) and the vector ((w's vector representation) - e_c)  (≈1 line)
        cosine_sim = cosine_similarity(e_b-e_a+e_c,word_to_vec_map[w])
        
        # If the cosine_sim is more than the max_cosine_sim seen so far,
            # then: set the new max_cosine_sim to the current cosine_sim and the best_word to the current word (≈3 lines)
        if cosine_sim > max_cosine_sim:
            max_cosine_sim = cosine_sim
            best_word = w
        ### END CODE HERE ###
        
    return best_word
1
2
3
triads_to_try = [('italy', 'italian', 'spain'), ('india', 'delhi', 'japan'), ('man', 'woman', 'boy'), ('small', 'smaller', 'large')]
for triad in triads_to_try:
    print ('{} -> {} :: {} -> {}'.format( *triad, complete_analogy(*triad,word_to_vec_map)))
italy -> italian :: spain -> spanish
india -> delhi :: japan -> tokyo
man -> woman :: boy -> girl
small -> smaller :: large -> larger

小结：

  • 计算两个向量之间的相似度可以采用余弦相似度
  • 在NLP中，第一步往往使用训练好的词向量

去除词向量中的歧视

首先计算词向量中关于性别的向量，可以用\(g = e_{woman}-e_{man}\)近似，当然，可以分别计算\(g_1 = e_{mother}-e_{father}\), \(g_2 = e_{girl}-e_{boy}\),，然后平均。本文采用简单的\(g = e_{woman}-e_{man}\)在编码空间中表示“相别”的概念。

1
2
g = word_to_vec_map['woman'] - word_to_vec_map['man']
print(g)
[-0.087144    0.2182     -0.40986    -0.03922    -0.1032      0.94165
 -0.06042     0.32988     0.46144    -0.35962     0.31102    -0.86824
  0.96006     0.01073     0.24337     0.08193    -1.02722    -0.21122
  0.695044   -0.00222     0.29106     0.5053     -0.099454    0.40445
  0.30181     0.1355     -0.0606     -0.07131    -0.19245    -0.06115
 -0.3204      0.07165    -0.13337    -0.25068714 -0.14293    -0.224957
 -0.149       0.048882    0.12191    -0.27362    -0.165476   -0.20426
  0.54376    -0.271425   -0.10245    -0.32108     0.2516     -0.33455
 -0.04371     0.01258   ]

下面分别计算不同的单词和向量\(g\)的余弦相似度。

1
2
3
4
5
6
7
print ('List of names and their similarities with constructed vector:')

# girls and boys name
name_list = ['john', 'marie', 'sophie', 'ronaldo', 'priya', 'rahul', 'danielle', 'reza', 'katy', 'yasmin']

for w in name_list:
    print (w, cosine_similarity(word_to_vec_map[w], g))
List of names and their similarities with constructed vector:
john -0.23163356146
marie 0.315597935396
sophie 0.318687898594
ronaldo -0.312447968503
priya 0.17632041839
rahul -0.169154710392
danielle 0.243932992163
reza -0.079304296722
katy 0.283106865957
yasmin 0.233138577679

可以看出，女性的名称和\(g\)的余弦相似度为正，男性的名称和\(g\)的余弦相似度为负。

下面再试试别的单词

1
2
3
4
5
print('Other words and their similarities:')
word_list = ['lipstick', 'guns', 'science', 'arts', 'literature', 'warrior','doctor', 'tree', 'receptionist', 
             'technology',  'fashion', 'teacher', 'engineer', 'pilot', 'computer', 'singer']
for w in word_list:
    print (w, cosine_similarity(word_to_vec_map[w], g))
Other words and their similarities:
lipstick 0.276919162564
guns -0.18884855679
science -0.0608290654093
arts 0.00818931238588
literature 0.0647250443346
warrior -0.209201646411
doctor 0.118952894109
tree -0.0708939917548
receptionist 0.330779417506
technology -0.131937324476
fashion 0.0356389462577
teacher 0.179209234318
engineer -0.0803928049452
pilot 0.00107644989919
computer -0.103303588739
singer 0.185005181365

可以发现，结果带有明显的性别歧视。例如，计算机和男性相近，而文学和女性相近。

线面采用Boliukbasi et al., 2016提出的方法减少这些向量的歧视性。注意到一些单词对，比如"actor"/"actress" 或者 "grandmother"/"grandfather"指定了性别，而其他的词，比如"receptionist" 或 "technology"，则是中性的，也即和性别无关。我们需要区别处理这两类单词。

对无性别意义的单词进行中性化处理

下图显示如何进行中性化处理。如果用50维空间进行词编码，该50维可以分为两部分：歧视方向\(g\)，和剩余的49维（或称为\(g_{\perp}\)）。在线性代数中，49维向量\(g_{\perp}\)和向量\(g\)正交。中性化处理是指，将词向量在\(g\)方向置零，只保留剩下的49维向量\(e_{receptionist}^{debiased}\)。

尽管\(g_{\perp}\)是49维的，但是为了显示方便，这里用1维的轴表示。

图 2: "receptionist"的词向量在中性化处理前后对比.

具体的计算步骤是：

\[e^{bias\_component} = \frac{e \cdot g}{||g||_2^2} * g\tag{2}\] \[e^{debiased} = e - e^{bias\_component}\tag{3}\]

实际上，上面的操作相当于将\(e\)投影到方向\(g\)上，得到\(e^{bias\_component}\)。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
def neutralize(word, g, word_to_vec_map):
    """
    Removes the bias of "word" by projecting it on the space orthogonal to the bias axis. 
    This function ensures that gender neutral words are zero in the gender subspace.
    
    Arguments:
        word -- string indicating the word to debias
        g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)
        word_to_vec_map -- dictionary mapping words to their corresponding vectors.
    
    Returns:
        e_debiased -- neutralized word vector representation of the input "word"
    """
    
    ### START CODE HERE ###
    # Select word vector representation of "word". Use word_to_vec_map. (≈ 1 line)
    e = word_to_vec_map[word]
    
    # Compute e_biascomponent using the formula give above. (≈ 1 line)
    e_biascomponent = np.dot(e, g)*g/np.sum(g**2)
 
    # Neutralize e by substracting e_biascomponent from it 
    # e_debiased should be equal to its orthogonal projection. (≈ 1 line)
    e_debiased = e - e_biascomponent
    ### END CODE HERE ###
    
    return e_debiased
1
2
3
4
5
e = "receptionist"
print("cosine similarity between " + e + " and g, before neutralizing: ", cosine_similarity(word_to_vec_map["receptionist"], g))

e_debiased = neutralize("receptionist", g, word_to_vec_map)
print("cosine similarity between " + e + " and g, after neutralizing: ", cosine_similarity(e_debiased, g))
cosine similarity between receptionist and g, before neutralizing:  0.330779417506
cosine similarity between receptionist and g, after neutralizing:  -8.52425655987e-17

对带性别意义的词中心化处理

去歧视也需要应用于像"actress" 和 "actor" 这样的单词对。该操作称为中心化，使得两个单词之间仅仅存在性别上的差异。一个具体的例子是：假如"actress" 和"babysit" 比"actor"和"babysit"之间更近。通过中性化处理，"babysit"和性别无关，但是无法保证"actor"和"actress" 到"babysit"的距离相等。中心化可以处理这个问题。

中心化的基本想法是，保证该对单词到49维向量\(g_\perp\)的距离相等。这样，也保证了这两个单词到\(e_{receptionist}^{debiased}\)的距离相等，或者到其他任何中性化之后的词相等。

具体的计算公式为：

\[ \mu = \frac{e_{w1} + e_{w2}}{2}\tag{4}\]

\[ \mu_{B} = \frac {\mu \cdot \text{bias_axis}}{||\text{bias_axis}||_2^2} *\text{bias_axis} \tag{5}\]

\[\mu_{\perp} = \mu - \mu_{B} \tag{6}\]

\[ e_{w1B} = \frac {e_{w1} \cdot \text{bias_axis}}{||\text{bias_axis}||_2^2} *\text{bias_axis} \tag{7}\] \[ e_{w2B} = \frac {e_{w2} \cdot \text{bias_axis}}{||\text{bias_axis}||_2^2} *\text{bias_axis} \tag{8}\]

\[e_{w1B}^{corrected} = \sqrt{ |{1 - ||\mu_{\perp} ||^2_2} |} * \frac{e_{\text{w1B}} - \mu_B} {|(e_{w1} - \mu_{\perp}) - \mu_B)|} \tag{9}\]

\[e_{w2B}^{corrected} = \sqrt{ |{1 - ||\mu_{\perp} ||^2_2} |} * \frac{e_{\text{w2B}} - \mu_B} {|(e_{w2} - \mu_{\perp}) - \mu_B)|} \tag{10}\]

\[e_1 = e_{w1B}^{corrected} + \mu_{\perp} \tag{11}\] \[e_2 = e_{w2B}^{corrected} + \mu_{\perp} \tag{12}\]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def equalize(pair, bias_axis, word_to_vec_map):
    """
    Debias gender specific words by following the equalize method described in the figure above.
    
    Arguments:
    pair -- pair of strings of gender specific words to debias, e.g. ("actress", "actor") 
    bias_axis -- numpy-array of shape (50,), vector corresponding to the bias axis, e.g. gender
    word_to_vec_map -- dictionary mapping words to their corresponding vectors
    
    Returns
    e_1 -- word vector corresponding to the first word
    e_2 -- word vector corresponding to the second word
    """
    
    ### START CODE HERE ###
    # Step 1: Select word vector representation of "word". Use word_to_vec_map. (≈ 2 lines)
    w1, w2 = pair
    e_w1, e_w2 = word_to_vec_map[w1],word_to_vec_map[w2]
    
    # Step 2: Compute the mean of e_w1 and e_w2 (≈ 1 line)
    mu = (e_w1 + e_w2)/2.0

    # Step 3: Compute the projections of mu over the bias axis and the orthogonal axis (≈ 2 lines)
    mu_B = np.dot(mu, bias_axis)*bias_axis/np.sum(bias_axis**2)
    mu_orth = mu - mu_B

    # Step 4: Use equations (7) and (8) to compute e_w1B and e_w2B (≈2 lines)
    e_w1B = np.dot(e_w1, bias_axis)*bias_axis/np.sum(bias_axis**2)
    e_w2B = np.dot(e_w2, bias_axis)*bias_axis/np.sum(bias_axis**2)
        
    # Step 5: Adjust the Bias part of e_w1B and e_w2B using the formulas (9) and (10) given above (≈2 lines)
    corrected_e_w1B = np.sqrt(np.abs(1.0-np.sum(mu_orth**2)))*(e_w1B-mu_B)/np.abs(e_w1-mu_orth-mu_B)
    corrected_e_w2B = np.sqrt(np.abs(1.0-np.sum(mu_orth**2)))*(e_w2B-mu_B)/np.abs(e_w2-mu_orth-mu_B)

    # Step 6: Debias by equalizing e1 and e2 to the sum of their corrected projections (≈2 lines)
    e1 = corrected_e_w1B + mu_orth
    e2 = corrected_e_w2B + mu_orth
                                                                
    ### END CODE HERE ###
    
    return e1, e2
1
2
3
4
5
6
7
8
print("cosine similarities before equalizing:")
print("cosine_similarity(word_to_vec_map[\"man\"], gender) = ", cosine_similarity(word_to_vec_map["man"], g))
print("cosine_similarity(word_to_vec_map[\"woman\"], gender) = ", cosine_similarity(word_to_vec_map["woman"], g))
print()
e1, e2 = equalize(("man", "woman"), g, word_to_vec_map)
print("cosine similarities after equalizing:")
print("cosine_similarity(e1, gender) = ", cosine_similarity(e1, g))
print("cosine_similarity(e2, gender) = ", cosine_similarity(e2, g))
cosine similarities before equalizing:
cosine_similarity(word_to_vec_map["man"], gender) =  -0.117110957653
cosine_similarity(word_to_vec_map["woman"], gender) =  0.356666188463

cosine similarities after equalizing:
cosine_similarity(e1, gender) =  -0.716572752584
cosine_similarity(e2, gender) =  0.739659647493

小结

  • 在NLP中，第一步往往是加载已经训练好的词编码模型
  • 相似度的计算方法很多，本文用的是余弦相似度
  • 词编码模型和图像处理中的自编码模型类似，都是将高维空间映射到低维特征空间
  • 去歧视在NLP中是一个不能忽视的工作，本文对性别的定义比较简单，可以进一步改进对性别的定义

参考资料