Activation Function

Activation Function

: layer에서 다음 layer로 전달할 때, 값이 과다/과소할 때 값을 조정하는 역할

 

Activatoin Function 종류

1. Sigmoid Function: 0~1 사이의 작은 실수 출력
2. ReLU(Rectified Linear Unit): 음수 → 0으로 반환
3. Linear: 1차 함수(Default)

 

⭐ output_dim activatoin

Sigmoid: 0~1 사이 값으로 반환되므로 이진 분류만에서 사용

ReLU: 음수값이 0으로 반환되므로, 값이 양수일경우에만 사용

 

# activation_function.py

import numpy as np

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score


# 1. Data
datasets = load_diabetes()
x = datasets.data
y = datasets.target

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    train_size=0.7
)


# 2. model
model = Sequential()
model.add(Dense(32, input_dim=10, activation='linear'))
model.add(Dense(16, activation='linear'))
model.add(Dense(1))


# 3. compile and train
model.compile(loss='mse', optimizer='adam')
model.fit(x_train, y_train, epochs=128, batch_size=32)


# 4. evaluate and predict
loss = model.evaluate(x_test, y_test)
y_predict = model.predict(x_test)

def RMSE (y_test, y_predict):
    return np.sqrt(mean_squared_error(y_test, y_predict))
# def 함수_이름(para1, para2):
    # return np.sqrt(mse)
print("RMSE: ", RMSE(y_test, y_predict))

r2 = r2_score(y_test, y_predict)
print("R2: ", r2)



'''
Result with Sigmoid
RMSE:  149.80864055376117
R2:  -2.883800175441581

Result with ReLU
RMSE:  56.906094021920666
R2:  0.41644697806090136

Result with Linear
MSE:  58.45104205742501
R2:  0.43929155713336476

'''

 

 

 

➕ Binary Classification에서 최종 layer에서 activation=’sigmoid’ 사용 시, 0~1 사이의 실수를 반환하므로 단순히 y값(0 또는 1인 정수)과 accuracy를 판단할 수 없는 오류 발생

⚠️ ValueError: Classification metrics can't handle a mix of binary and continuous targets

→ np.where()를 통해 조건식으로 0 또는 1 반환하여 acc 확인

# sigmoid_acc_cancer.py

import numpy as np

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense


# 1. Data
datasets = load_breast_cancer()

x = datasets['data'] # (569, 30) for training
y = datasets['target'] # (569, ) for predict

x_train, x_test, y_train, y_test = train_test_split(
    x, y,
    test_size=0.2,
    shuffle= True,
    random_state = 123
)


# 2. Model Construction
model = Sequential()
model.add(Dense(64, activation='linear', input_shape=(30,)))
model.add(Dense(32, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1, activation='sigmoid'))


# 3. Compile and train
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# Binary Classification Model Construction
# Last Activation=sigmoid, loss='binary_crossentropy'
# 이진분류에서는 정오 판단이 가능하므로 Accuracy 사용이 용이함

from tensorflow.keras.callbacks import EarlyStopping
earlyStopping = EarlyStopping(monitor='accuracy', mode='auto', patience=5, restore_best_weights=True, verbose=1)

hist = model.fit(x_train, y_train,
          epochs=100, batch_size=16,
          validation_split=0.2,
          callbacks=[earlyStopping],
          verbose=1)

'''
print(hist.history)
: model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) 기준으로 출력
-> 'loss', 'metrics-accuracy' 출력

'''


# 4. evaluate and predict
loss, accuracy = model.evaluate(x_test, y_test)
# return 값이 2개 이상일 경우, variable 2개 입력 가능

y_predict = model.predict(x_test) # y_test와 비교
# print(y_predict[:10]) # 10개 출력, 0<sigmoid<1
# print(y_test[:10]) # 10개 출력 y_test = 0 or 1
# 실수와 정수 Type Unmatching Error (ValueError: Classification metrics can't handle a mix of binary and continuous targets)

# Solve: ValueError(Classification metrics can't handle a mix of binary and continuous targets)
pred_class = np.where(y_predict >= 0.5, 1, 0) # 0.5 이상=1, 0.5 미만=0
# 조건에 따라 x or y에서 선택한 요소를 반환


from sklearn.metrics import r2_score, accuracy_score
acc = accuracy_score(y_test, pred_class)
print("accuarcy_score: ", acc)
# accuracy(x_test, pred_class)
# y_predict = model.predict(x_test)
# parameter가 다르므로 accuracy도 차이



'''
Result with converting binary classification from sigmoid
loss:  0.13962924480438232
accuracy:  0.9561403393745422
accuarcy_score:  0.956140350877193
accuracy와 accuarcy_score가 차이나는 이유: y_predict값을 0 또는 1로 변환 하였으므로

'''

 

소스 코드

🔗 HJ0216/TIL

 

참고 자료

📑 활성화 함수 (activation function)

📑 [Error] Tensorflow | 딥러닝 이진분류 | valueerror