이모저모 개발 블로그

기본 환경: IDE: VS code, Language: Python

⭐ 분류별 ouput_dim, activatoin, loss

# 이진분류: output_dim = (col), activatoin = 'sigmoid', loss = 'binarycrossentropy'
# 다중분류: output_dim = (class), activatoin = 'softmax', loss = 'categorical_crossentropy'

⭐ y data type이 분류에 속하는지 확인하는 방법

: np.unique(y) → return 값이 분류인지 확인

# 1. Data
datasets = load_wine()
x = datasets.data
y = datasets['target']

print(x.shape, y.shape) # (178, 13) (178,)
# print(y)를 통해서 0, 1, 2...로 이뤄져있으면 분류
# (문제) 데이터가 많을 때 판단이 어려울수 있음
# (해결) np.unique
print(np.unique(y, return_counts=True))
# [0 1 2] 중복값을 제외한 y 값, return_counts를 통해 element의 개수 반환
# output_dim = 1, y_class = 3

⚠️ data(y)는 col=1, class = 3([0 1 2])

1. 데이터의 특성이 총 3개([0 1 2])이지만 col=1이므로 class 수만큼 col을 늘려줘야함

2. 데이터를 0, 1, 2 등으로 수치화하였을 때, 다음과 같은 오해의 소지가 발생할 수 있음

OneHotEncoding

: 10진 정수 형식을 특수한 2진 binary 형식으로 변환하는 것

: col을 class 수만큼 늘리고, 수치적 특성이 없는 데이터임을 보일 수 있음

⭐ One-Hot Encoding 종류

1. sklearn: one_hot encoding

# sklearn: one_hot encoding

onehot_encoder=OneHotEncoder(sparse=False)
reshaped=y.reshape(len(y), 1)
onehot=onehot_encoder.fit_transform(reshaped)
print(onehot)
# [[1. 0. 0.]...[0. 0. 1.]]
print(onehot.shape) (150, 3)

2. keras: to_categorical

# keras: to_categorical

to_cat=to_categorical(y)
print(to_cat)
# [[1. 0. 0.]...[0. 0. 1.]]
print(to_cat.shape) (150, 3)

3. pandas: get_dummies

# pandas: get_dummies

print(pd.get_dummies(y))
[150 rows x 3 columns]

⚠️ One-Hot Encoding 후 발생한 타입 불일치 오류

ValueError: Classification metrics can't handle a mix of multilabel-indicator and continuous-multioutput targets

'''
y_test
[[1. 0. 0.]
 [0. 0. 1.]
 [1. 0. 0.]
 [0. 1. 0.]
 [0. 1. 0.]]

 y_predict
[[9.9925297e-01 7.4696867e-04 2.5099497e-13] 0
 [4.1013454e-10 2.7499644e-03 9.9725002e-01] 2
 [9.9945968e-01 5.4027850e-04 1.0871933e-13] 0
 [2.5293427e-06 6.0845017e-01 3.9154729e-01] 1
 [6.0919424e-06 8.0725497e-01 1.9273894e-01]] 1


Summary
y_test: [1. 0. 0.] 타입
y_predict: 실수 타입
(문제) 타입 불일치 오류
ValueError: Classification metrics can't handle a mix of multilabel-indicator and continuous-multioutput targets

(해결) y_predict = np.argmax(y_predict, axis=1)
argmax를 사용해서 arg 중 가장 큰 값을 뽑아 위치값을 반환

(최종 결과)
[0 2 0 2 1 1 0 2 0 2 2 2 2 0 0 0 2 0 2 1 0 2 1 1 0 2 1 1 1 2]
[0 2 0 1 1 1 0 2 0 2 2 2 2 0 0 0 2 0 2 1 0 2 1 1 0 2 1 1 1 1]
를 기준으로 accuracy 판단
'''

➕ fetch_covtype과 함께하는 One-Hot Encoding 응용

1. sklearn: one_hot encoding

문제1: ValueError: Expected 2D array, got 1D array instead: array=[5 5 2 ... 3]
OneHotEncoder는 Matrix 2차원만 입력받음
해결1: Matrix형태로 reshape → reshape: y = y.reshape(581012, 1)

문제2: TypeError: A sparse matrix was passed, but dense data is required.
train_test_splite를 사용할 수 있는 자료구조: numpy.ndarray
OneHotEncoding 후 자료구조: scipy.sparse._csr.csr_matrix
해결2: Use X.toarray() to convert to a dense numpy array.→ y = y.toarray()

# oneHotEncoder_fetch_covtype.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.datasets import fetch_covtype
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder
from sklearn.metrics import accuracy_score


# 1. Data
datasets = fetch_covtype()
x = datasets.data # (581012, 54)
y = datasets['target'] # (581012,)

y = y.reshape(581012, 1)
# 문제1: ValueError: Expected 2D array, got 1D array instead: array=[5 5 2 ... 3]
# OneHotEncoder는 Matrix 2차원만 입력받음
# 해결1: Matrix형태로 reshape → reshape: y = y.reshape(581012, 1)

ohe = OneHotEncoder()
ohe.fit(y)
y = ohe.transform(y)
print(type(y)) # <class 'scipy.sparse._csr.csr_matrix'>
y = y.toarray()
print(type(y)) # <class 'numpy.ndarray'>
# 문제2: TypeError: A sparse matrix was passed, but dense data is required.
# train_test_splite를 사용할 수 있는 자료구조: numpy.ndarray
# OneHotEncoding 후 자료구조: scipy.sparse._csr.csr_matrix
# 해결2: Use X.toarray() to convert to a dense numpy array.→ y = y.toarray()

x_train, x_test, y_train, y_test = train_test_split(
    x,y,
    shuffle=True,
    random_state=333,
    test_size=0.2,
    stratify=y
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, activation='relu', input_shape=(54, )))
model.add(Dense(64, activation='sigmoid'))
model.add(Dense(32,activation='relu'))
model.add(Dense(16,activation='linear'))
model.add(Dense(7,activation='softmax'))


# 3. Compile and train
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy']
              )

model.fit(x_train, y_train, epochs=256, batch_size=128,
          validation_split=0.2,
          verbose=1)


# 4. evaluate and predict
loss, accuracy = model.evaluate(x_test, y_test)

y_predict = model.predict(x_test)
y_predict = np.argmax(y_predict, axis=1)
y_test = np.argmax(y_test, axis=1)

acc = accuracy_score(y_test, y_predict)
print("accuracy_score: ", acc)


'''
Result
loss:  0.596627414226532
accuracy:  0.7471494078636169
accuracy_score:  0.7471493851277505

'''

2. keras: to_categorical

문제: class 값이 0부터 시작하지 않으면 class 처음에 0 추가
해결: np.delete() 활용

# to_categorical_fetch_covtype.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.utils import to_categorical

from sklearn.datasets import fetch_covtype
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# 1. Data
datasets = fetch_covtype()
x = datasets.data
y = datasets['target']

y=to_categorical(y)
'''
print(type(y)) # 타입 확인 <class 'numpy.ndarray'>

문제
to_catergorical: class가 0부터 시작하지 않을 때, 앞에 0을 추가 -> 추가된 0만큼의 자원의 낭비가 발생
to_categorical(y): (0,1,2,3,4,5,6,7)
y: (1,2,3,4,5,6,7)

해결
y = np.delete(y, 0, 1)
np.delete(array, idx, axis)
열 -> 축(axis)이 지정되지 않으면 1차원으로 변환된(flatten) array에서 지정한 인덱스 값 제거
np.delete(a, 1, axis=1)과 같이 축을 지정: 축을 따라 지정한 인덱스의 서브어레이를 제거한 어레이를 반환

np.delete 참조: 
numpy_delete.py

'''

y = np.delete(y, 0, axis=1)

x_train, x_test, y_train, y_test = train_test_split(
    x,y,
    shuffle=True,
    random_state=333,
    test_size=0.2,
    stratify=y
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, activation='relu', input_shape=(54, )))
model.add(Dense(64, activation='sigmoid'))
model.add(Dense(32,activation='relu'))
model.add(Dense(16,activation='linear'))
model.add(Dense(7,activation='softmax'))


# 3. Compile and train
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy']
              )

model.fit(x_train, y_train, epochs=100, batch_size=128,
          validation_split=0.2,
          verbose=1)


# 4. evaluate and predict
loss, accuracy = model.evaluate(x_test, y_test)

y_predict = model.predict(x_test)
y_predict = np.argmax(y_predict, axis=1)
y_test = np.argmax(y_test, axis=1)

acc = accuracy_score(y_test, y_predict)
print("accuracy_score: ", acc)



'''
Result
accuracy_score:  0.729817646704474

'''

3. pandas: get_dummies

문제: type이 get_dummies 처리 후, (numpy → pandas) → value가 잘못 인식됨
해결: type 재변환: pandas → numpy

# get_dummies_fetch_covtype.py

import numpy as np
import pandas as pd

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.datasets import fetch_covtype
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score


# 1. Data
datasets = fetch_covtype()
x = datasets.data
y = datasets['target']

y = pd.get_dummies(y)
'''
y = pd.get_dummies(y): idx 및 head 출력
부분 출력: print(y[:10])
print(y)
        1  2  3  4  5  6  7
0       0  0  0  0  1  0  0
1       0  0  0  0  1  0  0
2       0  1  0  0  0  0  0
3       0  1  0  0  0  0  0
4       0  0  0  0  1  0  0
...    .. .. .. .. .. .. ..
581007  0  0  1  0  0  0  0
581008  0  0  1  0  0  0  0
581009  0  0  1  0  0  0  0
581010  0  0  1  0  0  0  0
581011  0  0  1  0  0  0  0
[581012 rows x 7 columns]

print(type(y)) 
<class 'pandas.core.frame.DataFrame'>


Error
ValueError: Shape of passed values is (116203, 1), indices imply (116203, 7)
추가하고자 하는 값은 116203개의 값을 7개의 열에 추가해주려고 하는데, 정작 입력한 값은 116203개의 값을 1개의 열 값

Value가 잘못 인식된 이유:
get dummies를 통해서 y: (numpy-> pandas) -> np method에 pandas 바로 입력하면 해당 error 발생
해결: pandas -> numpy로 바꿔주기
'''

y = y.to_numpy() # pandas -> numpy
# y = y.values # pandas -> numpy

x_train, x_test, y_train, y_test = train_test_split(
    x,y,
    shuffle=True,
    random_state=333,
    test_size=0.2,
    stratify=y
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, activation='relu', input_shape=(54, )))
model.add(Dense(64, activation='sigmoid'))
model.add(Dense(32,activation='relu'))
model.add(Dense(16,activation='linear'))
model.add(Dense(7,activation='softmax'))


# 3. Compile and train
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy']
              )

model.fit(x_train, y_train, epochs=256, batch_size=64,
          validation_split=0.2,
          verbose=1)


# 4. evaluate and predict
loss, accuracy = model.evaluate(x_test, y_test)

y_predict = model.predict(x_test)
y_predict = np.argmax(y_predict, axis=1)
y_test = np.argmax(y_test, axis=1)

acc = accuracy_score(y_test, y_predict)
print("accuracy_score: ", acc)



'''
Result
loss:  0.5851762294769287
accuracy:  0.7449549436569214
accuracy_score:  0.7449549495279812

'''

소스 코드

🔗 HJ0216/TIL

참고 자료

📑 원 핫 인코딩(One-Hot Encoding) 이해하기

📑 Numpy.Delete: How To Remove Elements From A NumPy Array

기본 환경: IDE: VS code, Language: Python

⭐ plt.plot 형태에 따른 모델의 훈련 과정 확인

# plot_boston.py

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split


# 1. Data
datasets = load_boston()
x = datasets.data # (506, 13)
y = datasets.target # (506, )

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    test_size=0.2,
    shuffle= True,
    random_state = 333
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, input_shape=(13,)))
model.add(Dense(32))
model.add(Dense(16))
model.add(Dense(1))


# 3. Compile and train
model.compile(loss='mse', optimizer='adam')
hist = model.fit(x_train, y_train,
          epochs=256,
          batch_size=32,
          validation_split=0.2,
          verbose=0)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)

print(hist.history)
'''
fit의 history는 loss 결과값을 dictionary(key-value(list)) 형태로 반환
key: 'loss', 'val_loss' / value = list[] 형태의 epochs의 loss, val_loss 값들의 집합
{'loss': [25004.84765625, 1219.100830078125, 160.86378479003906, 82.83763122558594, 73.0763931274414, 71.3211669921875, 71.86249542236328, 70.77513885498047, 68.52639770507812, 68.08159637451172],
'val_loss': [2787.144775390625, 272.2074279785156, 78.57952880859375, 58.332862854003906, 55.535221099853516, 54.82481002807617, 54.39116668701172, 56.427764892578125, 59.47801971435547, 55.58904266357422]}
'''
print(hist.history['loss']) # key -> value 반환
print(hist.history['val_loss'])


import matplotlib.pyplot as plt
# (epochs, loss)의 산점도 및 그래프를 작성할 수 있음

plt.figure(figsize=(9, 6)) # 그래프 사이즈 설정: figsize=(가로 inch, 세로 inch) 
plt.plot(hist.history['loss'], c='red', marker='.', label='loss') # x 추론 가능 시, x 생략 가능
plt.plot(hist.history['val_loss'], c='blue', marker='.', label='val_loss')
# c: color, marker: graph 형태, label: graph name

plt.title('Boston loss') # graph name
plt.xlabel('epochs') # x축 이름
plt.ylabel('loss') # x축 이름
plt.grid() # 격자 표시
plt.legend(loc='upper right') # 선 이름(label) 표시, location 미 지정 시 그래프와 안 겹치게 생성
plt.show()



'''
Result
loss 26.94292640686035

plt.show()
문제: min(loss)가 아닌 훈련값(Overfit)이 존재 
해결: min(loss)가 아닌 지점에서 훈련 중지

'''

⚠️ plot graph를 통해 val_loss가 높아지는 비효율적인 훈련이 이뤄지는 구간을 확인

⭐ EarlyStopping을 사용하여, 성능에 안 좋아질 경우에 훈련을 자동으로 종료

⭐ EarlyStopping Mechanism
→ 매 훈련마다 loss 확인
→ min loss가 나오지 않더라도 넘어갈 수 있는 횟수 지정
예: 5번까지 min loss가 나오지 않더라도 training을 지속
(⭐ 최적의 early stopping timing을 찾는 것이 중요)
→ 횟수 내에 min loss가 안 나오면 훈련(fit) 수행 중지

# earlyStopping_boston.py

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.callbacks import EarlyStopping

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split


# 1. Data
datasets = load_boston()
x = datasets.data # (506, 13)
y = datasets.target # (506, )

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    test_size=0.2,
    shuffle= True,
    random_state = 333
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, input_shape=(13,)))
model.add(Dense(32))
model.add(Dense(16))
model.add(Dense(1))


# 3. Compile and train
model.compile(loss='mse', optimizer='adam')

earlyStopping = EarlyStopping(monitor='val_loss', mode='min', patience=16, restore_best_weights=True, verbose=1)
# monitor: val_loss로 earlyStopping 지점 확인
# mode: accuracy-max, loss-min, max인지 min인지 모를 때, auto 사용
# patience=5: 갱신이 되지 않더라도 16번 참음
# restore_best_weight: 훈련이 끊긴 지점이 아닌 최적의 weight 지점을 저장
# verbose를 통해 earlyStopping 지점을 확인할 수 있음: Restoring model weights from the end of the best epoch.

hist = model.fit(x_train, y_train,
          epochs=256,
          batch_size=16,
          validation_split=0.2,
          callbacks=[earlyStopping],
          # 정지된 지점-5: min(val_loss)
          # 문제: 5번 인내 후, 최소가 아닌 val_loss 지점에서의 weight가 아닌 끊긴 지점에서의 weight가 반환
          # 해결: restore_best_weights="True"를 통해 최적의 weight 지점을 반환
          # restore_best_weights="False" Defualt
          # 최적의 weight로 predict 수행(false일 경우, epoch가 마무리된 weight를 기준으로 predict 수행)
          verbose=1
          )


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)



'''
Result

Epoch 79/256
21/21 [==============================] - 0s 6ms/step - loss: 66.6039 - val_loss: 83.2419
Restoring model weights from the end of the best epoch: 63.
Epoch 00079: early stopping

'''

➕ plt에 한글 사용 방법

# matplotlib 한글 설정

from matplotlib import font_manager, rc
font_path = "C:\Windows\Fonts\malgun.ttf"
font = font_manager.FontProperties(fname=font_path).get_name()
rc('font', family=font)

➕ epoch 당 loss 값의 산점도(Scatter)

# scatter_boston.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split


# 1. Data
datasets = load_boston()
x = datasets.data # (506, 13)
y = datasets.target # (506, )

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    test_size=0.2,
    shuffle= True,
    random_state = 123
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, input_shape=(13,)))
model.add(Dense(32))
model.add(Dense(16))
model.add(Dense(1))


# 3. Compile and train
model.compile(loss='mse', optimizer='adam')
hist = model.fit(x_train, y_train,
          epochs=256,
          batch_size=16,
          validation_split=0.2,
          verbose=0)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)


import matplotlib.pyplot as plt

x_len = np.arange(len(hist.history['loss']))
plt.scatter(x_len, hist.history['loss'])
# len(hist.history['loss']): fit(x_train)의 loss의 개수 = epochs
# 산점도(x 생략 불가)
plt.show() # 작성한 plt show

 

 

➕ Model Check Point

⭐ EarlyStopping: 최적의 weight가 갱신이 안되면 훈련을 끊어주는 역할
⭐ ModelCheckPoint: 최적의 weight가 갱신될 때마다 저장해주는 역할

 

1. Save the Model Check Point

# save_ModelCheckPoint.py

import numpy as np

from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.metrics import mean_squared_error, r2_score

path = './_save/'


# 1. Data
dataset = load_boston()
x = dataset.data # for training
y = dataset.target # for predict

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    train_size=0.7,
    random_state=123
)

scaler = StandardScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)


# 2. Model(Function)
input1 = Input(shape=(13,))
dense1 = Dense(64, activation='relu')(input1)
dense2 = Dense(64, activation='sigmoid')(dense1)
dense3 = Dense(32, activation='relu')(dense2)
dense4 = Dense(32, activation='linear')(dense3)
output1 = Dense(1, activation='linear')(dense4)
model = Model(inputs=input1, outputs=output1)


# 3. compile and train
model.compile(loss='mse', optimizer='adam', metrics=['mae'])

earlyStopping = EarlyStopping(monitor='val_loss', mode='min', patience=32, restore_best_weights=True, verbose=1)

modelCheckPoint = ModelCheckpoint(monitor='val_loss', mode='auto', verbose=1,
                                   save_best_only=True,
                                   filepath=path+'MCP/save_ModelCheckPoint.hdf5') # MCP/ = MCP파일 하단
# 가중치 및 모델 저장 확장자: h5
# ModelCheckPoint 저장 확장자: hdf5

model.fit(x_train, y_train,
          epochs=512,
          batch_size=16,
          validation_split=0.2,
          callbacks=[earlyStopping, modelCheckPoint],
          verbose=1)

'''
epochs=512
Epoch 00001: val_loss improved from inf to 481.11652, saving model to ./_save/MCP\save_ModelCheckPoint1.hdf5
-> 처음 훈련은 최상의 결과값이므로 저장
Epoch 00002: val_loss improved from 481.11652 to 273.11346, saving model to ./_save/MCP\save_ModelCheckPoint1.hdf5
-> 2번째 훈련 개선 -> 덮어쓰기
-> 반복
Epoch 00024: val_loss did not improve from 10.37429
-> 개선되지 않을 경우 저장 X
-> 개선되지 않은 결과가 20번 반복될 경우, EarlyStopping = 가장 성능이 좋은 ModelCheckPoint 지점
Epoch 67/512
14/18 [======================>.......] - ETA: 0s - loss: 5.7881 - mae: 1.6581
Restoring model weights from the end of the best epoch: 35.

MCP 저장
MSE:  4.433804789382322
R2:  0.7567847452094035

EarlyStopping: 최적의 weight가 갱신이 안되면 훈련을 끊어주는 역할
ModelCheckPoint: 최적의 weight가 갱신될 때마다 저장해주는 역할

'''


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)
y_predict = model.predict(x_test)

def RMSE (y_test, y_predict):
    return np.sqrt(mean_squared_error(y_test, y_predict))
print("RMSE: ", RMSE(y_test, y_predict))

r2 = r2_score(y_test, y_predict)
print("R2: ", r2)



'''
Result
MSE:  4.433804789382322
R2:  0.7567847452094035

'''

 

2. Load the Model Check Point

# load_ModelCheckPoint.py

import numpy as np

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, StandardScaler
from sklearn.metrics import mean_squared_error, r2_score

from tensorflow.keras.models import Sequential, Model, load_model
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint

path = './_save/'


# 1. Data
dataset = load_boston()
x = dataset.data # for training
y = dataset.target # for predict

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    train_size=0.7,
    random_state=123
)

scaler = StandardScaler()
scaler.fit(x_train)
x_train = scaler.transform(x_train)
x_test = scaler.transform(x_test)

# 2. Model(Function)


# 3. compile and train
model = load_model(path+'MCP/save_ModelCheckPoint.hdf5') # load_model로 모델과 가중치 불러오기
'''
MCP 저장
MSE:  4.433804789382322
R2:  0.7567847452094035
'''


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)
y_predict = model.predict(x_test)

def RMSE (y_test, y_predict):
    return np.sqrt(mean_squared_error(y_test, y_predict))
print("RMSE: ", RMSE(y_test, y_predict))

r2 = r2_score(y_test, y_predict)
print("R2: ", r2)



'''
Result
RMSE:  4.433804789382322
R2:  0.7567847452094035

'''

⭐ Training Result 비교

1. restore_best_weights = True

(Break 지점이 아닌 최적의 weight를 저장)

load_model(EarlyStopping)
load_model(ModelCheckPoint)
→ 연산 결과 모두 동일

2. restore_best_weights = False    
load_model(EarlyStropping)
load_model(ModelCheckPoint)
→ 연산 결과 ModelCheckPoint 결과가 더 우수함
(restore_best_weights = False, 최적 weight에서 연산을 멈추지 않고 patience만큼 연산이 밀린 후 연산이 멈추므로)
⚠️ 단, train data set에서 최적의 weight 지점이 test data set에서는 아닐 수 있듯, 밀려난 지점이 test data set에서는 더 우수한 weight를 뽑아낼 수 있음

 

 

 

소스 코드

🔗 HJ0216/TIL

 

참고 자료

📑 [python] matplotlib로 플롯 그릴 때 한글 깨짐 문제 해결 방법 (윈도우)

📑 Keras - Epoch와 오차(Loss)간 관게를 그래프로 확인하기

 

기본 환경: IDE: VS code, Language: Python

Model Construction 후, Training → Evaluation → Predict 과정을 거침

➕ Training → 'Validation' → Evaluation → Predict

: 검증 과정을 추가하여, 훈련의 체계성과 예측 가능성을 높임
(예: 학습 후, 문제 풀이 과정 추가)

⚠️ Validation을 실시한다고 평가 예측 결과가 무조건 우수해지는 것은 아님

⭐ Validation Data split 방법

1. 직접 나누기(x_val, y_val)

# validation_data_make.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense


# 1. Data
# For train
x_train = np.array(range(1, 11)) # 1,2,3,4,5,6,7,8,9,10
y_train = np.array(range(1, 11))

# For evaluate
x_test = np.array([11,12,13])
y_test = np.array([11,12,13])

# For validatoin
x_validation = np.array([14,15,16])
y_validation = np.array([14,15,16])


# 2. Model
model = Sequential()
model.add(Dense(32, input_dim=1, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1))


# 3. compile and train
model.compile(loss='mse', optimizer='adam')
model.fit(x_train, y_train, epochs=128, batch_size=32,
          validation_data=(x_validation, y_validation))
# validation_data를 통해서 val_loss 추가
# 훈련 + '검증(Validation)' + 평가 (fit + 'validation'+ evaluate)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)

result = model.predict([17])
print("predict [17]: ", result)



'''
Result(make val_data)
training loss - loss: 0.5571, val_loss: 5.2239
evaluate loss - loss: 2.8699
predict [17] - [14.324451]

'''

2. 전체 데이터에서 slice 활용

# validation_data_slice.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense


# 1. Data
x = np.array(range(1, 17)) # 1~16
y = np.array(range(1, 17))

x_train = x[:10] # 1~10
y_train = y[:10]
x_test = x[10:13] # 11~13
y_test = y[10:13]
x_validation = x[13:] # 14~16
y_validation = y[13:]
# slicing [초과:이하]


# 2. Model
model = Sequential()
model.add(Dense(32, input_dim=1, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1))


# 3. compile and train
model.compile(loss='mse', optimizer='adam')
model.fit(x_train, y_train, epochs=128, batch_size=32,
          validation_data=(x_validation, y_validation))
# validation_data를 통해서 val_loss 추가
# 훈련 + '검증(Validation)' + 평가 (fit + 'validation'+ evaluate)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)

result = model.predict([17])
print("predict [17]: ", result)



'''
Result(slice val_data)
training loss - loss: 0.1758, val_loss: 0.0382
evaluate loss - loss: 0.0025
predict [17] - [16.688446]

'''

3. train_test_split() 활용

# validation_data_split.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.model_selection import train_test_split


# 1. Data
x = np.array(range(1, 17))
y = np.array(range(1, 17))


x_train, x_test_tmp, y_train, y_test_tmp = train_test_split(
    x, y,
    shuffle=False,
    train_size=0.625,
    random_state=123
)

x_test, x_val, y_test, y_val = train_test_split(
    x_test_tmp, y_test_tmp,
    shuffle=False,
    train_size=0.5,
    random_state=123
)


# 2. Model
model = Sequential()
model.add(Dense(32, input_dim=1, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1))


# 3. compile and train
model.compile(loss='mse', optimizer='adam')
model.fit(x_train, y_train, epochs=128, batch_size=32,
          validation_data=(x_val, y_val))
# validation_data를 통해서 val_loss 추가
# 훈련 + '검증(Validation)' + 평가 (fit + 'validation'+ evaluate)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)

result = model.predict([17])
print("predict [17]: ", result)



'''
Result(split val_data)
training loss - loss: 0.0505, val_loss: 0.2850
evaluate loss - loss: 0.1105
predict [17] - [16.333452]

'''

4. fit()에서 val_split 활용

# validation_data_split(fit).py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.model_selection import train_test_split


# 1. Data
x = np.array(range(1, 17))
y = np.array(range(1, 17))

x_train, x_test, y_train, y_test = train_test_split(
    x,y,
    test_size=0.2,
    random_state=1234
)


# 2. Model
model = Sequential()
model.add(Dense(32, input_dim=1, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1))


# 3. compile and train
model.compile(loss='mse', optimizer='adam')
model.fit(x_train, y_train, epochs=128, batch_size=32,
          validation_split=0.25)
# validation_data를 통해서 val_loss 추가
# 훈련 + '검증(Validation)' + 평가 (fit + 'validation'+ evaluate)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)

result = model.predict([17])
print("predict [17]: ", result)



'''
Result(split(fit) val_data)
training loss - loss: 0.1299, val_loss: 0.1981
evaluate loss - loss: 0.2424
predict [17] - [16.16812]

'''

소스 코드

🔗 HJ0216/TIL

Activation Function

: layer에서 다음 layer로 전달할 때, 값이 과다/과소할 때 값을 조정하는 역할

Activatoin Function 종류

1. Sigmoid Function: 0~1 사이의 작은 실수 출력
2. ReLU(Rectified Linear Unit): 음수 → 0으로 반환
3. Linear: 1차 함수(Default)

⭐ output_dim activatoin

Sigmoid: 0~1 사이 값으로 반환되므로 이진 분류만에서 사용

ReLU: 음수값이 0으로 반환되므로, 값이 양수일경우에만 사용

# activation_function.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score


# 1. Data
datasets = load_diabetes()
x = datasets.data
y = datasets.target

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    train_size=0.7
)


# 2. model
model = Sequential()
model.add(Dense(32, input_dim=10, activation='linear'))
model.add(Dense(16, activation='linear'))
model.add(Dense(1))


# 3. compile and train
model.compile(loss='mse', optimizer='adam')
model.fit(x_train, y_train, epochs=128, batch_size=32)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)
y_predict = model.predict(x_test)

def RMSE (y_test, y_predict):
    return np.sqrt(mean_squared_error(y_test, y_predict))
# def 함수_이름(para1, para2):
    # return np.sqrt(mse)
print("RMSE: ", RMSE(y_test, y_predict))

r2 = r2_score(y_test, y_predict)
print("R2: ", r2)



'''
Result with Sigmoid
RMSE:  149.80864055376117
R2:  -2.883800175441581

Result with ReLU
RMSE:  56.906094021920666
R2:  0.41644697806090136

Result with Linear
MSE:  58.45104205742501
R2:  0.43929155713336476

'''

➕ Binary Classification에서 최종 layer에서 activation=’sigmoid’ 사용 시, 0~1 사이의 실수를 반환하므로 단순히 y값(0 또는 1인 정수)과 accuracy를 판단할 수 없는 오류 발생

⚠️ ValueError: Classification metrics can't handle a mix of binary and continuous targets

→ np.where()를 통해 조건식으로 0 또는 1 반환하여 acc 확인

# sigmoid_acc_cancer.py

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense


# 1. Data
datasets = load_breast_cancer()

x = datasets['data'] # (569, 30) for training
y = datasets['target'] # (569, ) for predict

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    test_size=0.2,
    shuffle= True,
    random_state = 123
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, activation='linear', input_shape=(30,)))
model.add(Dense(32, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='sigmoid'))


# 3. Compile and train
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Binary Classification Model Construction
# Last Activation=sigmoid, loss='binary_crossentropy'
# 이진분류에서는 정오 판단이 가능하므로 Accuracy 사용이 용이함

from tensorflow.keras.callbacks import EarlyStopping
earlyStopping = EarlyStopping(monitor='accuracy', mode='auto', patience=5, restore_best_weights=True, verbose=1)

hist = model.fit(x_train, y_train,
          epochs=100, batch_size=16,
          validation_split=0.2,
          callbacks=[earlyStopping],
          verbose=1)

'''
print(hist.history)
: model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) 기준으로 출력
-> 'loss', 'metrics-accuracy' 출력

'''


# 4. evaluate and predict
loss, accuracy = model.evaluate(x_test, y_test)
# return 값이 2개 이상일 경우, variable 2개 입력 가능

y_predict = model.predict(x_test) # y_test와 비교
# print(y_predict[:10]) # 10개 출력, 0<sigmoid<1
# print(y_test[:10]) # 10개 출력 y_test = 0 or 1
# 실수와 정수 Type Unmatching Error (ValueError: Classification metrics can't handle a mix of binary and continuous targets)

# Solve: ValueError(Classification metrics can't handle a mix of binary and continuous targets)
pred_class = np.where(y_predict >= 0.5, 1, 0) # 0.5 이상=1, 0.5 미만=0
# 조건에 따라 x or y에서 선택한 요소를 반환


from sklearn.metrics import r2_score, accuracy_score
acc = accuracy_score(y_test, pred_class)
print("accuarcy_score: ", acc)
# accuracy(x_test, pred_class)
# y_predict = model.predict(x_test)
# parameter가 다르므로 accuracy도 차이



'''
Result with converting binary classification from sigmoid
loss:  0.13962924480438232
accuracy:  0.9561403393745422
accuarcy_score:  0.956140350877193
accuracy와 accuarcy_score가 차이나는 이유: y_predict값을 0 또는 1로 변환 하였으므로

'''

소스 코드

🔗 HJ0216/TIL

참고 자료

📑 활성화 함수 (activation function)

📑 [Error] Tensorflow | 딥러닝 이진분류 | valueerror

⭐ 1. Git bash에서 remote repository 변경 전 pull을 통해 버전 맞추기

$ git pull origin main
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 9 (delta 6), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (9/9), 2.10 KiB | 46.00 KiB/s, done.
From https://github.com/HJ0216/TIL
 * branch            main       -> FETCH_HEAD
   1843ac3..8797bb6  main       -> origin/main
Updating 1843ac3..8797bb6
Fast-forward
 README.md | 3 +++
 1 file changed, 3 insertions(+)

2. remote repository 내 불필요한 파일 제거

$ git rm --cached -r test.txt
rm 'test.txt'

# remote repository 내 불필요한 test.txt 파일 제거

폴더 제거 시,

git rm --cached -r 폴더명

# rm: remove
# --cached: 원격 저장소의 폴더 또는 파일 삭제 옵션
# --cached가 없을 경우, 로컬 저장소의 폴더 또는 파일도 삭제
# -r: directory 옵션

파일 제거 시,

git rm --cached -r 파일명
git rm --cached 파일명

# 둘 중 택 1
# -r 옵션은 폴더를 지울 때는 필수, 파일을 지울 때는 선택

3. 버전 관리를 위한 commit 진행

$ git commit -m "remove a test.txt"
[main 759bd78] remove a test.txt
 1 file changed, 1 deletion(-)
 delete mode 100644 test.txt

4. 원격 저장소로 push

$ git push -u origin main
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 223 bytes | 223.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/HJ0216/TIL.git
   8797bb6..759bd78  main -> main
branch 'main' set up to track 'origin/main'.

➕ --cached를 입력하지 않았을 때, remote repository와 local repository

1. 임의 파일(will_be_deleted.txt) 생성

2. git bash에서 add, commit, push 진행

$ git add /c/git_total/will_be_deleted.txt


$ git commit -m "add a test file for git pratice"
[main ed8b4a2] add a test file for git pratice
 1 file changed, 1 insertion(+)
 create mode 100644 will_be_deleted.txt


$ git push -u origin main
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 340 bytes | 340.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
To https://github.com/HJ0216/TIL.git
   567d81d..ed8b4a2  main -> main
branch 'main' set up to track 'origin/main'.

3. remote repository에서 push 내용 확인

4. git bash에서 --cached 제외하고 파일 remove 진행

$ git rm will_be_deleted.txt
rm 'will_be_deleted.txt'

$ git commit -m "remove a test file for git pratice"
[main 95467dc] remove a test file for git pratice
 1 file changed, 1 deletion(-)
 delete mode 100644 will_be_deleted.txt

$ git push -u origin main
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 231 bytes | 231.00 KiB/s, done.
Total 2 (delta 1), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To https://github.com/HJ0216/TIL.git
   ed8b4a2..95467dc  main -> main
branch 'main' set up to track 'origin/main'.

5. remote repository 및 local repository에서 push 내용 확인

참고 자료

📑 [git] github에 잘못 올라간 폴더 및 파일 삭제하기