數據量太大，怎樣用python壹次讀取壹個手寫體mnist

mnist的結構如下，選取train-images

TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type] [value] [description]

0000 32 bit integer 0x00000803(2051) magic number

0004 32 bit integer 60000 number of images

0008 32 bit integer 28 number of rows

0012 32 bit integer 28 number of columns

0016 unsigned byte ? pixel

0017 unsigned byte ? pixel

........

xxxx unsigned byte ? pixel

也就是之前我們要讀取4個 32 bit integer

試過很多方法，覺得最方便的，至少對我來說還是使用

struct.unpack_from()

filename = 'train-images.idx3-ubyte'

binfile = open(filename , 'rb')

buf = binfile.read()()

先使用二進制方式把文件都讀進來

index = 0

magic, numImages , numRows , numColumns = struct.unpack_from('>IIII' , buf , index)

index += struct.calcsize('>IIII')

然後使用struc.unpack_from

'>IIII'是說使用大端法讀取4個unsinged int32

然後讀取壹個圖片測試是否讀取成功

im = struct.unpack_from('>784B' ,buf, index)

index += struct.calcsize('>784B')

im = np.array(im)

im = im.reshape(28,28)

fig = plt.figure()

plotwindow = fig.add_subplot(111)

plt.imshow(im , cmap='gray')

plt.show()

'>784B'的意思就是用大端法讀取784個unsigned byte

完整代碼如下

import numpy as np

import struct

import matplotlib.pyplot as plt

filename = 'train-images.idx3-ubyte'

binfile = open(filename , 'rb')

buf = binfile.read()

index = 0

magic, numImages , numRows , numColumns = struct.unpack_from('>IIII' , buf , index)

index += struct.calcsize('>IIII')

im = struct.unpack_from('>784B' ,buf, index)

index += struct.calcsize('>784B')

im = np.array(im)

im = im.reshape(28,28)

fig = plt.figure()

plotwindow = fig.add_subplot(111)

plt.imshow(im , cmap='gray')

plt.show()

只是為了測試是否成功所以只讀了壹張圖片