Python之DataFrame数据处理
|Word count:1.3k|Reading time:7min|Post View:
Python 之 DataFrame 数据处理
1. 说明
DataFrame 是 Pandas 库中处理表的数据结构，可看作是 python
中的类似数据库的操作，是 Python 数据挖掘中最常用的工具。下面介绍
DataFrame 的一些常用方法。
2. 遍历
1) 代码
1
2
3
4
5
6
7
8
| import pandas as pd
import math
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3],'data2':[4,5,6]})
print(df)
for idx,item in df.iterrows():
print(idx)
print(item)
|
2) 结果
1
2
3
4
5
6
7
8
9
10
| data1 data2 key
0 1 4 a
1 2 5 b
2 3 6 c
0
data1 1
data2 4
key a
Name: 0, dtype: object
… 略
|
3. 同时遍历两个数据表
1) 代码
1
2
3
4
5
6
7
8
9
10
| import pandas as pd
import math
df1=pd.DataFrame({'key':['a','b'],'data1':[1,2]})
df2=pd.DataFrame({'key':['c','d'],'data2':[4,5]})
for (idx1,item1),(idx2,item2) in zip(df1.iterrows(),df2.iterrows()):
print("idx1",idx1)
print(item1)
print("idx2",idx2)
print(item2)
|
2) 结果
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| ('idx1', 0)
data1 1
key a
Name: 0, dtype: object
('idx2', 0)
data2 4
key c
Name: 0, dtype: object
('idx1', 1)
data1 2
key b
Name: 1, dtype: object
('idx2', 1)
data2 5
key d
Name: 1, dtype: object
|
4. 取一行或多行
1) 代码
1
2
3
4
5
6
| import pandas as pd
import math
df1=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
df2=df1[:1]
print(df2)
|
2) 结果
5. 取一列或多列
1) 代码
1
2
3
4
5
6
7
| import pandas as pd
import math
df1=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
df2=pd.DataFrame()
df2['key2']=df1['key']
print(df2)
|
2) 结果
6. 列连接（横向：变宽）：merge
1) 代码
1
2
3
4
5
| import pandas as pd
df1=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
df2=pd.DataFrame({'key':['a','b','c'],'data2':[4,5,6]})
df3=pd.merge(df1,df2)
|
2) 结果
1
2
3
4
5
6
7
8
9
10
11
12
| data1 key
0 1 a
1 2 b
2 3 c
data2 key
0 4 a
1 5 b
2 6 c
data1 key data2
0 1 a 4
1 2 b 5
2 3 c 6
|
7. 行连接（纵向：变长）：concat
1) 代码
1
2
3
4
5
| import pandas as pd
df1=pd.DataFrame({'key':['a','b','c'],'data':[1,2,3]})
df2=pd.DataFrame({'key':['d','e','f'],'data':[4,5,6]})
df3=pd.concat([df1,df2])
|
2) 结果
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| data key
0 1 a
1 2 b
2 3 c
data key
0 4 d
1 5 e
2 6 f
data key
0 1 a
1 2 b
2 3 c
0 4 d
1 5 e
2 6 f
|
8. 对某列做简单变换
1) 代码
1
2
3
4
5
6
7
|
import pandas as pd
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
print(df)
df['data1']=df['data1']+1
print(df)
|
2) 结果
1
2
3
4
5
6
7
8
| data1 key
0 1 a
1 2 b
2 3 c
data1 key
0 2 a
1 3 b
2 4 c
|
9. 对某列做复杂变换
1) 代码
1
2
3
4
5
6
7
| import pandas as pd
import math
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
print(df)
df['data1']=df['data1'].apply(lambda x: math.sin(x))
print(df)
|
2) 结果
1
2
3
4
5
6
7
8
| data1 key
0 1 a
1 2 b
2 3 c
data1 key
0 0.841471 a
1 0.909297 b
2 0.141120 c
|
10. 对某列做函数处理
1) 代码
1
2
3
4
5
6
7
8
9
10
11
| import pandas as pd
def testme(x):
print("???",x)
y = x + 3000
return y
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
print(df)
df['data1']=df['data1'].apply(testme)
print(df)
|
2) 结果
1
2
3
4
5
6
7
8
9
10
11
| data1 key
0 1 a
1 2 b
2 3 c
('???', 1)
('???', 2)
('???', 3)
data1 key
0 3001 a
1 3002 b
2 3003 c
|
11. 用某几列计算生成新列
1) 代码
1
2
3
4
5
6
| import pandas as pd
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3],'data2':[4,5,6]})
print(df)
df['data3']=df['data1']+df['data2']
print(df)
|
2) 结果
1
2
3
4
5
6
7
8
| data1 data2 key
0 1 4 a
1 2 5 b
2 3 6 c
data1 data2 key data3
0 1 4 a 5
1 2 5 b 7
2 3 6 c 9
|
12. 用某几列用函数生成新列
1) 代码
1
2
3
4
5
6
7
8
9
10
11
| import pandas as pd
import math
def testme(x):
print(x['data1'],x['data2'])
return x['data1'] + x['data2']
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3],'data2':[4,5,6]})
print(df)
df['data3']=df.apply(testme, axis=1)
print(df)
|
2) 结果
1
2
3
4
5
6
7
8
9
10
11
| data1 data2 key
0 1 4 a
1 2 5 b
2 3 6 c
(1, 4)
(2, 5)
(3, 6)
data1 data2 key data3
0 1 4 a 5
1 2 5 b 7
2 3 6 c 9
|
13. 删除列
1) 代码
1
2
3
4
5
6
7
| import pandas as pd
import math
df=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3],'data2':[4,5,6]})
print(df)
df=df.drop(['data2'],axis=1)
print(df)
|
2) 结果
1
2
3
4
5
6
7
8
| data1 data2 key
0 1 4 a
1 2 5 b
2 3 6 c
data1 key
0 1 a
1 2 b
2 3 c
|
14. One-Hot
变换（把一列枚举型变为多列数值型）
1) 代码
1
2
3
4
5
6
7
8
9
| import pandas as pd
import math
df1=pd.DataFrame({'key':['a','b','c'],'data1':[1,2,3]})
print(df1)
df2=pd.get_dummies(df1['key'])
print(df2)
df3=pd.get_dummies(df1)
print(df3)
|
2) 结果
1
2
3
4
5
6
7
8
9
10
11
12
| data1 key
0 1 a
1 2 b
2 3 c
a b c
0 1 0 0
1 0 1 0
2 0 0 1
data1 key_a key_b key_c
0 1 1 0 0
1 2 0 1 0
2 3 0 0 1
|
15. 其它常用方法
1) 求均值方差，中位数等
df[f].describe()
2) 求均值
df[f].mean()
3) 求方差
df[f].std()
4) 清除空值
df.dropna()
5) 填充空值
df.fillna()
